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EXECUTIVE  SUMMARY 


DoD-Centric  and  Independent  Technology  Evaluation  Capability  (DITEC)  users  may  decide  that 
certain  capability  examinations  are  less  meaningful  for  them  than  others.  This  report  introduces  a 
mechanism  for  allowing  security  non-experts  define  their  situational  needs  and  be  matched  with  the 
technology,  or  suite  of  technologies,  that  best  satisfy  them. 
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INTRODUCTION 


The  United  States  has  been  the  victim  of  a  number  of  attacks  on  both  privately  held  and  publicly  owned 
networks  (e.g.,  the  Sony  Pictures  hack  which  was  attributed  to  North  Korea  [Sanger  and  Perlroth,  2014]  and 
the  Office  of  Personnel  Management  (OPM)  breaches  which  compromised  the  security  of  more  than 
22  million  current  or  former  federal  employees  (Nakashima,  2015).  It  has  been  estimated  that  the  United 
States  faces  more  than  5,000  cyberattacks  every  hour,  and  the  Pentagon  is  attacked  10  million  times  each 
day  (NetStandard,  2014).  In  light  of  the  growing  number  and  catastrophic  results  of  cyberattacks, 
cybersecurity  has  become  one  of  the  Department  of  Defense’s  (DoD)  top  priorities.  The  marketplace  of 
cybersecurity  technologies  is  filled  with  many  different  products,  all  making  competing  claims  of 
capabilities.  Until  recently,  there  has  been  no  uniform  system  for  evaluating  cybersecurity  products  and 
services. 

The  Space  and  Naval  Warfare  Systems  Center  Pacific’s  (SSC  Pacific)  IA  Division  has  developed  the 
DoD-Centric  and  Independent  Technology  Evaluation  Capability  (DITEC)  to  streamline  cybersecurity 
technology  evaluation.  Specifically,  DITEC  defines  a  process  for  evaluating  whether  or  not  a  product  meets 
DoD  needs,  security  metrics  for  measuring  how  well  those  needs  are  met,  and  a  framework  for  comparing 
various  products  that  address  the  same  cybersecurity  technology  area  (Romero-Mariona  ,  2014).  DITEC 
evaluates  cybersecurity  products  and  services  at  three  levels  of  granularity:  Capability,  Sub-Capability,  and 
Sub-Capability  Element.  There  are  10  capabilities,  with  each  capability  having  a  varying  number  of  sub¬ 
capabilities  and  sub-capability  elements.  There  are  44  sub-capabilities  and  109  sub-capability  elements  that 
are  evaluated.  DITEC  also  features  an  ability  that  allows  individual  users  to  prioritize  certain  capabilities  or 
sub-capabilities  and  provide  weighted  averages  that  better  match  the  user’s  needs  to  technology  evaluations 
(Hallman,  Romero-Mariona,  Kline;  and  San  Miguel,  2014). 

Because  of  the  growth  in  cybersecurity  workforces  and  a  marketplace  flooded  with  diverse  technologies, 
it  is  expected  that  cybersecurity  professionals  may  not  have  great  knowledge  of  all  cybersecurity 
technologies.  The  DITEC  Technology  Matching  Tool  (TMT)  was  designed  to  aid  technical  and  acquisition 
personnel  in  selecting  the  appropriate  cybersecurity  technology  to  meet  their  needs.  DITEC’s  Sub- 
Capability  level  of  granularity  is  at  a  level  of  specificity  that  can  be  used  to  point  the  user  towards  certain 
cybersecurity  technologies  over  others.  The  TMT  leads  the  user  through  a  series  of  yes/no  questions  for 
each  of  the  109  sub-capability  elements,  which  gives  a  binary  “User  Requirement”  vector.  The  TMT 
utilizes  the  User  Priority  Designation  (UPD)  scheme  described  by  Hallman  et  al.  (2014)  to  scale  the  User 
Requirement  sub-vectors,  where  the  sub-vectors  correspond  to  DITEC’s  sub-capabilities  (Table  1).  Each 
cybersecurity  technology  is  also  given  a  binary  vector  based  on  whether  or  not  it  performs  each  sub¬ 
capability  element.  The  scaled  user  requirements  vector  and  each  cybersecurity  technology  vector  are  then 
compared  with  the  1  2  norm.  The  technologies  are  then  matched  to  the  user’s  requirements,  ordered  from  the 
least  1  2  norm  to  greatest. 
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Table  1.  DITEC  sub-capabilities. 


Priority 

UPD  Rank 

Scaling  Factor 

Top  priority 

5 

1.00 

High  priority 

4 

0.85 

Medium  priority 

3 

0.55 

Low  priority 

2 

0.25 

Minimal  priority 

1 

0.10 

No  priority 

0 

0.00 

PROTOTYPING  AND  IMPLEMENTING  THE  TMT 

A  prototype  of  the  DITEC  TMT  was  built  using  MATLAB®.  This  prototype  TMT  leads  the  user  through 
a  series  of  yes/no  questions  for  each  of  DITEC’s  44  sub-capabilities,  assigning  to  each  sub-capability  a 
numerical  priority.  (Figure  1)  The  User  Requirement  vectors  tested  were  cybersecurity  technology  vectors, 
for  this  demonstration  the  binary  vector  for  High  Assurance  Internet  Protocol  Encryptor  (HAIPE) 
technologies. 


Figure  1 .  The  DITEC  TMT  leading  the  user  through  prioritization  of  each  sub-capability  technology  vectors, 
for  this  demonstration  the  binary  vector  for  HAIPE  technologies. 

There  are  two  Technology  Tables  of  results  currently  given  by  the  TMT.  The  first  Technology  Table 
gives  the  user  a  simple  comparison  of  the  User  Requirement  vector  to  each  cybersecurity  technology  vector. 
(Figure  2)  The  second  Technology  Table  creates  a  sub- vector  from  the  cybersecurity  technology  and  User 
Requirement  vectors  based  on  whether  an  element  in  the  prioritized  User  Requirement  vector  is  unequal  to 
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zero.  If  an  element  in  the  prioritized  User  Requirement  vector  is  unequal  to  zero,  then  that  element  and  its 
corresponding  element  of  each  cybersecurity  technology  vector  are  transferred  to  a  new  vector  for 
comparison.  That  is,  the  first  Technology  Table  only  takes  into  consideration  the  sub-capabilities  to  which 
the  user  has  assigned  a  positive  UPD  Rank  or  a  Sub-Capability  Element  for  which  the  user  has  explicitly 
affirmed  a  need.  (Figure  3). 


Command  Window 

New  to  MATLAB?See  resources  for  Gettina  Started. 

X 

Tech_table  = 

Technol ogy 

Best_fit 

1 HAIPEs 1 

0 

'Application' 

0 

'Storage  Device' 

0 

'IP  Encryption' 

1.4142 

1  Remediation' 

2.2361 

'Digital  Forensics' 

2.4495 

' Reporting' 

2.4495 

'Load  Balancing' 

2.4495 

'  Sniffer' 

2.6458 

'RAIDS  System' 

2.6458 

'Device  Scanner' 

2.6458 

'Application  Scanner' 

2.6458 

'Vulnerability  Scanner' 

2.6458 

' Network-based' 

2.8284 

' Host-based' 

2.8284 

' Appl i cati on-based ' 

2.8284 

'Security  Information  and  Event  Management  (SIEM)  systems' 

2.8284 

'Compliance  Checking' 

2.8284 

'Fault  Management' 

2.8284 

'Performance  Management' 

2.8284 

'Storage  and  Disaster  Recovery' 

2.8284 

'Anti  Malware  and  Program  Control' 

2.8284 

'  Policy  Definition' 

3 

'Security  Management' 

3 

' Network-based2' 

3 

' Host-based2' 

3 

' Appl i cati on-based ' 

3 

'Operating  System  Security  Management' 

3 

'Aplication  Security  Management' 

3 

_ 

'Configuration  Management' 

3.1623 

Based  on  the  user  requirements  provided,  the  following  table  provides  a  list  of  "best  fit"  technology. 

The  technologies  are  listed  from  best  fit  (top)  to  least  best  fit 

(bottom) . 

Tech_table_2  = 

Technol ogy 

New_metric 

'HAIPEs' 

0 

'IP  Encryption' 

1.7321 

' Host-based2' 

1.7321 

' Appl i cati on-based ' 

2 

'Performance  Management' 

2 

'Security  Management' 

2 

fx  'Load  Balancing' 

2 

2 

Figure  2.  The  DITEC  TMT  showing  unprioritized  results  when  the  User  Requirement  vector  is  the  HAIPE 
technology  vector. 


Figures  2  and  3  show  the  difference  in  TMT  Technology  Tables  when  the  sub-capability  elements  are 
unsealed.  (That  is,  the  scaling  factor  is  1.00.)  Figure  2  shows  the  Technology  Table  of  comparisons  to  the 
full  User  Requirement  vector  and  each  cybersecurity  technology  vector.  Figure  3  shows  the  Technology 
Table  of  comparisons  when  only  sub-vector  of  positively  affirmed  sub-capability  elements  of  the  User 
Requirement  vector  and  corresponding  cybersecurity  technology  vectors  are  considered.  In  this 
demonstration,  the  HAIPE  technology  vector  was  chosen  as  the  User  Requirement  vector,  and  as  expected, 
HAIPE  is  the  top  match  in  both  Technology  Tables.  Note  the  differences  in  TMT  results  further  down  in  the 
technology  tables.  In  Figure  2,  the  second,  third,  and  fourth  matches  are  Application  Security  technologies, 
Storage  Device  technologies,  and  IP  Encryption  technologies.  In  Figure  3,  on  the  other  hand,  the  second, 
third,  and  fourth  matches  are  IP  Encryption  technologies,  Host-Based  Security  technologies,  and 
Application-based  technologies.  Because  Technology  Table  2  takes  into  account  only  those  specific 
capabilities  that  the  user  affirms,  other  capabilities  that  the  user  may  not  be  interested  in  are  disregarded  and 
the  user  is  better  matched  to  the  cybersecurity  technologies  meeting  their  needs. 
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Figure  3.  The  DITEC  TMT  showing  unprioritized  results  when  the  User  Requirement  vector  is  the  HAIPE 
technology  vector  and  only  the  sub-capability  elements  which  the  user  affirms  are  considered. 

Where  Figures  2  and  3  show  the  unprioritized  results,  the  UPD  integrated  TMT  allows  the  user  446 
prioritizations  with  which  they  may  better  match  their  unique  circumstances  to  an  IA  technology.  Figure  4 
in  the  current  example  of  the  MATLAB®  prototype  shows  the  TMT  Technology  Table  1  resulting  from  one 
possible  priority  profile  and  the  same  User  Requirement  vector,  while  Figure  5  shows  the  corresponding 
Technology  Table  2.  Even  with  the  prioritized  scaling,  Technology  Table  1  gives  similar  results  to  the 
unsealed  TMT  in  Figure  2.  Note  that  with  the  user  priorities  scaling  the  User  Requirement  vector,  the  TMT 
gives  slightly  different  results  in  the  second  Technology  Table.  Even  though  the  User  Requirement  vector 
was  the  HAIPE  technology  vector,  the  user’s  priorities  scaled  the  vector  in  such  a  way  that  a  HAIPE  may 
not  be  the  best  choice  for  their  needs  and  priorities.  Rather,  the  user  has  prioritized  sub-capabilities  in  such 
a  way  that  Storage  Device  technologies  meet  their  affirmed  capability  needs. 
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Command  Window 


New  to  MATLAJ3?  See  resources  for  Gettina  Started. 

X 

Tech_table  = 

Technol ogy 

Best_fit 

1 HAIPEs 1 

0 

'Application' 

0 

'Storage  Device1 

0 

'IP  Encryption' 

1.4142 

'  Remediation' 

2.2361 

'Digital  Forensics' 

2.4495 

' Reporting' 

2.4495 

'Load  Balancing1 

2.4495 

'  Sniffer' 

2.6458 

'RAIDS  System' 

2.6458 

'Device  Scanner' 

2.6458 

'Application  Scanner' 

2.6458 

'Vulnerability  Scanner' 

2.6458 

' Network-based' 

2.8284 

' Host-based' 

2.8284 

' Appl i cati on-based ' 

2.8284 

'Security  Information  and  Event  Management  (SIEM)  systems' 

2.8284 

'Compliance  Checking' 

2.8284 

'Fault  Management' 

2.8284 

'Performance  Management' 

2.8284 

'Storage  and  Disaster  Recovery' 

2.8284 

'Anti  Malware  and  Program  Control' 

2.8284 

'  Policy  Definition' 

3 

'Security  Management' 

3 

' Network-based2' 

3 

' Host-based2' 

3 

' Appl i cati on-based ' 

3 

'Operating  System  Security  Management' 

3 

'Aplication  Security  Management' 

3 

_ 

'Configuration  Management' 

3.1623 

Based  on  the  user  requirements  provided,  the  following  table  provides  a  list  of  "best  fit"  technology. 

The  technologies  are  listed  from  best  fit  (top)  to  least  best  fit 

(bottom) . 

Tech_table_2  = 

Technol ogy 

New_metric 

'Storage  Device' 

1 .8802 

'HAIPEs' 

2.5367 

'Application' 

2.5367 

' Host-based' 

2.5367 

'IP  Encryption' 

2.7083 

' Appl i cati on-based ' 

2.7991 

fx  'Network-based' 

2.8169 

Figure  4.  The  DITEC  TMT  showing  prioritized  results  when  the  User  Requirement  vector  is  the  HAIPE 
technology  vector. 
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Command  Window 


® 


New  to  MATLAB?  See  resources  for  Gettina  Started. 

X 

1  UU  1  L  i  iui  iuywmwTTC““ 

'Performance  Management' 

2.8284 

'Storage  and  Disaster  Recovery' 

2.8284 

'Anti  Malware  and  Program  Control' 

2.8284 

'Policy  Definition' 

3 

'Security  Management' 

3 

' Network-based2' 

3 

' Host-based2' 

3 

' Appl i cati on-based ' 

3 

'Operating  System  Security  Management' 

3 

' Application  Security  Management' 

3 

'Configuration  Management' 

3.1623 

Based  on  the  user  requirements  provided,  the  following  table  provides  a  list  of  "best  fit"  technology. 

The  technologies  are  listed  from  best  fit  (top)  to  least  best  fit 

(bottom) . 

Tech_table_2  = 

Technol ogy 

Neiv_metric 

'Storage  Device' 

1.8802 

' HAIPEs' 

2.5367 

'Application' 

2.5367 

' Host-based' 

2.5367 

'IP  Encryption' 

2.7083 

' Appl i cati on-based ' 

2.7991 

' Network-based' 

2.8169 

'  Sniffer' 

2.8522 

'Policy  Definition' 

2.9724 

' Host-based2' 

3.0224 

'Device  Scanner' 

3.0389 

' Appl i cati on-based ' 

3.1361 

'Performance  Management' 

3.1361 

'Security  Management' 

3.1361 

'Load  Balancing' 

3.1361 

'Application  Scanner' 

3.1361 

'Vulnerability  Scanner' 

3.1361 

'Operating  System  Security  Management' 

3.1361 

'Aplication  Security  Management' 

3.1361 

'Anti  Malware  and  Program  Control' 

3.1361 

'Compliance  Checking' 

3.152 

'Fault  Management' 

3.152 

'RAIDS  System' 

3.1992 

'Configuration  Management' 

3.2917 

' Reporting' 

3.3068 

'Storage  and  Disaster  Recovery' 

3.3369 

' Network-based2' 

3.4547 

'Security  Information  and  Event  Management  (SIEM)  systems' 

3.4547 

'Digital  Forensics' 

3.4547 

'  Remediation' 

3.4547 

f\  » 

- 

Figure  5.  The  DITEC  TMT  showing  prioritized  results  when  the  User  Requirement  vector  is  the  HAIPE 
technology  vector  and  only  the  sub-capability  elements  which  the  user  affirms  are  considered. 


INTEGRATING  THE  TMT  INTO  DITEC 

The  TMT  application  within  DITEC  was  written  in  the  Python™  programming  language  and  takes 
advantage  of  the  Django®  web  application  framework.  Using  Django’s  form  wizard,  the  user  is  guided 
through  a  series  of  steps  to  determine  the  right  technology  and  suite  of  technologies  to  meet  their  needs. 
Each  step  in  the  tool  represents  a  technological  capability  and  prompts  the  user  to  select  a  rating  from  0  to  5 
for  each  of  the  capability’s  sub  capabilities.  These  ratings  are  compared  to  each  technology  to  determine  the 
“best  fit”  technologies  and  technology  suites.  Django’s  form  wizard  is  an  application  that  splits  forms 
across  multiple  web  pages.  It  maintains  user  data  in  the  back  end  for  processing  after  the  final  step  in  the 
wizard  is  completed  and  provides  data  validation  between  steps.  The  TMT  queries  the  base  capability  table 
and  generates  a  form  for  each  capability.  Each  form  is  composed  of  a  title  with  capability  information  and  a 
list  of  fields  which  represent  the  sub  capabilities  associated  to  the  capability.  Each  field  is  composed  of  a 
label  for  the  sub  capability,  a  slider  and  an  integer  box.  The  user  can  select  a  rating  for  the  slider  by 
dragging  the  slider  bar  from  0  to  5.  The  integer  box  will  update  when  the  slider  value  is  changed  and  vice- 
versa. 
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A  back  button  is  provided  which  will  take  the  user  back  to  the  previous  step.  A  next  button  is  provided 
which  will  validate  the  users  ratings  before  continuing  to  the  next  step.  Finally,  a  skip  button  is  provided 
which  will  set  the  rating  for  each  field  to  0  and  continue  to  the  next  step.  Currently,  the  TMT  wizard 
displays  steps  with  fields  at  the  sub  capability  level.  However,  the  data  maintained  in  the  wizard  is  at  the 
sub  capability  element  level.  User  ratings  are  simply  passed  down  from  sub-capability  to  sub-capability 
element.  Data  is  stored  in  the  form  of  a  dictionary  with  sub-capability  elements  as  the  key  and  the  users’ 
ratings  as  the  value  and  is  maintained  in  a  session  on  a  per-site-visitor  basis.  Tooltips  are  provided  when  the 
user  hovers  the  mouse  over  a  capability,  sub-capability,  or  technology  item  in  the  user  interface.  A  tool  tip 
is  tied  to  the  description  column  in  the  corresponding  items  table.  The  order  of  the  steps  in  the  wizard  is 
determined  by  the  TMT  sort  order  column  in  the  base  Capability  table. 
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